Resolves the 3 failing CI jobs and all 30 inline review comments on
PR #24 (and the predecessor PR #14, whose comments were ported via the
branch rename `dra` → `feature/shahid-velero-restic`).

CI failures fixed
-----------------
* Helm Template Validation — chart no longer declares an upstream
  `velero` subchart dependency (the chart provides its OWN Velero +
  Restic templates; the two would collide and `helm template` could
  not run because `helm dependency build` is not invoked in CI).
* Helm Lint — removed the duplicate `cluster-backup.restoreName`
  template definition in `_helpers.tpl`.
* Documentation Validation / Markdown Lint — `cluster-backup/README.md`
  now satisfies MD031/MD022/MD032 (blank lines around fences /
  headings / lists). Added `cluster-backup/CHANGELOG.md` with a
  `[1.0.0] - 2026-05-22` entry that matches `Chart.yaml`'s
  `version: 1.0.0` — required by the tightened version-consistency
  check landed in #21.
* trailing-whitespace / end-of-file-fixer — stripped trailing whitespace
  from every cluster-backup yaml + shell file and ensured every file
  ends with `\n`.

Security & correctness (per Copilot / @romandidomizio review)
------------------------------------------------------------
* `automountServiceAccountToken: false` flipped to `true` on BOTH the
  Velero server and the Restic node-agent ServiceAccounts. Both
  controllers MUST call the Kubernetes API (Velero for cross-namespace
  resource enumeration / Backup-Restore CRDs; Restic for
  PodVolumeBackup / PodVolumeRestore / ResticRepository / coordination
  leases). Disabling token automount without a projected-token
  workaround silently broke the controllers.
* Removed `helm/templates/velero-secret.yaml`. The chart was rendering
  a placeholder Secret named `cluster-backup-cloud-credentials` which
  collided with the real Secret that `deploy.sh` creates out-of-band
  (`helm upgrade` would fail with "resource already exists" on every
  re-install). The Secret is now solely owned by `deploy.sh`.
  Removed the corresponding `checksum/secret` annotation from the
  Velero Deployment and Restic DaemonSet.
* Added `helm/templates/velero-service.yaml`. The chart previously
  shipped a `ServiceMonitor` selecting a Service that did not exist,
  and `NOTES.txt` instructed users to
  `kubectl port-forward svc/<fullname>-velero` — also missing.
  Service is `ClusterIP`, metrics-only.
* `helm/templates/networkpolicy.yaml` egress no longer uses `to: []`
  for any rule. DNS is constrained to the kube-system namespace;
  external S3 traffic is allowed to `0.0.0.0/0` MINUS RFC1918,
  link-local, and loopback ranges (so restic / velero can never
  accidentally exfiltrate to in-cluster IPs over an S3-shaped path);
  Kubernetes API access is constrained to the `default` namespace
  ClusterIP.
* `helm/templates/restic-daemonset.yaml` no longer mounts host `/` via
  `hostPath`. Restic only needs read access to `/var/lib/kubelet/pods`
  (the standard kubelet pod-volume root), so that's the only host
  mount that remains. Mounting host-root was incompatible with Pod
  Security `restricted` and not required for restic operation.
* `helm/templates/backup-schedules.yaml` passes a full
  `dict "Chart" $.Chart "Release" $.Release "Values" $.Values
   "Template" $.Template ...` context into the schedule-naming and
  schedule-config helpers, removing any ambiguity about what
  `cluster-backup.fullname` and `cluster-backup.labels` resolve to
  when called from within a `range` loop body.
* `helm/templates/_helpers.tpl`: simplified
  `cluster-backup.serviceAccountName` and
  `cluster-backup.resticServiceAccountName` to return deterministic
  names. The previous form gated on `.Values.velero.{server,restic}
  .serviceAccount.create`, which is not a path that exists in
  `values.yaml` — it was a remnant of the now-removed `velero`
  subchart pattern. The helpers always returned `"default"`, which
  conflicted with the ClusterRoleBinding subject the chart actually
  installed; Velero would have been denied every API call.
* `helm/templates/NOTES.txt` rewritten: `kubectl create backup …` /
  `kubectl create restore …` are NOT valid kubectl subcommands for
  Velero CRDs and were misleading users into thinking the controller
  was broken when in reality they were running a non-existent
  kubectl verb. NOTES now uses the Velero CLI (`velero backup get`,
  `velero schedule get`, etc.) and points readers at the install
  docs. Same fix applied across README.md, deploy.sh, verify.sh,
  test-local.sh, and (post-deployment) usage messages.

Script hardening
----------------
* `deploy.sh`:
  - Switched to `#!/usr/bin/env bash` + `set -euo pipefail`.
  - All temporary files now live in a single `mktemp -d`-created
    directory with mode 0700, cleaned up via an `EXIT|INT|TERM` trap.
    The previous version wrote `/tmp/s3-credentials` and
    `/tmp/cluster-backup-values.yaml` at fixed paths — world-readable
    on some hosts and racy under concurrent runs.
  - S3 secret-key prompt uses `read -rs` (no echo). The credential is
    piped into `kubectl create secret … --from-file=cloud=/dev/stdin
    --dry-run=client -o yaml | kubectl apply -f -`, so the secret
    never lives on disk and never appears in argv (`ps`).
  - All configuration accepts env-var inputs (`TENANT`, `CLUSTER`,
    `ENVIRONMENT`, `S3_*`) so the script is usable in CI/automation
    without an interactive TTY.
* `test-local.sh`:
  - MinIO image pinned (`minio/minio:RELEASE.2024-08-17T01-24-54Z`).
    Was `minio/minio:latest`, which made local test runs
    non-reproducible and exposed them to silent upstream breaking
    changes.
  - MinIO credentials no longer hardcoded as `minioadmin / minioadmin`
    in the manifest. Access key defaults to `localdev-access`; secret
    key is a freshly-generated 24-char random value per run
    (overridable via `TEST_MINIO_{ACCESS,SECRET}_KEY` env vars).
    Rendered into the Secret over a stdin pipe — same no-argv-leak
    pattern as deploy.sh.
  - Service now correctly exposes BOTH the S3 API (NodePort 30000)
    AND the web console (NodePort 30001). The original Service mapped
    only 9000 to NodePort 30000 yet the "MinIO Console" line pointed
    at 30000 — that's the S3 API, not the console. Web console
    address is now `http://localhost:30001`.
  - Replaced fixed `sleep 30` waits with `wait_for_velero_phase` which
    polls `.status.phase` of the Backup / Restore CR and emits
    `kubectl describe` + `velero backup logs` diagnostics on timeout.
  - Uses the Velero CLI for backup / restore creation.
* `verify.sh`:
  - Same shebang + strict-mode + Velero-CLI updates as deploy.sh.
  - `test_backup_creation` polls phase with diagnostics instead of
    blind-sleeping; also skips gracefully if the Velero CLI isn't
    installed locally (verify is run on hosts that may not have it).
* `create-tenant-cluster.sh`:
  - All upstream components are version-pinned via env-var defaults:
    `DOKS_K8S_VERSION`, `INGRESS_NGINX_VERSION`, `CERT_MANAGER_VERSION`,
    `METRICS_SERVER_VERSION`. The previous version used
    `doctl kubernetes cluster create … --version latest` and
    `metrics-server/releases/latest/`, making provisioning silently
    non-reproducible.
  - Team-member namespace is normalized to RFC1123
    (`AnnaF` → `annaf`, etc.) before being passed to `kubectl create
    namespace`. K8s rejects uppercase-containing namespace names with
    an opaque error, and the previous version of this script would
    silently fail at `create_team_member_namespace` for any contributor
    whose handle wasn't already lowercase-alphanumeric.
  - `create_team_member_access` no longer reads `.secrets[0].name` to
    harvest a long-lived ServiceAccount token (the BoundServiceAccount-
    Token feature in K8s ≥1.24 means that field is empty on modern
    clusters, so the function was returning an empty kubeconfig).
    Switched to `kubectl create token` (TokenRequest API) with an
    explicit 24h duration. Kubeconfig is written with `umask 077` so
    the file containing the bearer token is mode 0600.

Not in scope for this commit
----------------------------
* The Restic DaemonSet still ships with a `SYS_ADMIN` capability add
  and a non-root securityContext (uid 65534). Real-world restic
  operation typically requires either root or a relaxed
  podSecurityContext to read pod volumes owned by arbitrary uids;
  tightening this further requires a per-environment decision about
  the Pod Security Standard for the `velero` namespace. Flagged in
  `cluster-backup/CHANGELOG.md` under "Notes for operators".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Resolves the 3 failing CI jobs and all 30 inline review comments on
PR #24 (and the predecessor PR #14, whose comments were ported via the
branch rename `dra` → `feature/shahid-velero-restic`).

CI failures fixed
-----------------
* Helm Template Validation — chart no longer declares an upstream
  `velero` subchart dependency (the chart provides its OWN Velero +
  Restic templates; the two would collide and `helm template` could
  not run because `helm dependency build` is not invoked in CI).
* Helm Lint — removed the duplicate `cluster-backup.restoreName`
  template definition in `_helpers.tpl`.
* Documentation Validation / Markdown Lint — `cluster-backup/README.md`
  now satisfies MD031/MD022/MD032 (blank lines around fences /
  headings / lists). Added `cluster-backup/CHANGELOG.md` with a
  `[1.0.0] - 2026-05-22` entry that matches `Chart.yaml`'s
  `version: 1.0.0` — required by the tightened version-consistency
  check landed in #21.
* trailing-whitespace / end-of-file-fixer — stripped trailing whitespace
  from every cluster-backup yaml + shell file and ensured every file
  ends with `\n`.

Security & correctness (per Copilot / @romandidomizio review)
------------------------------------------------------------
* `automountServiceAccountToken: false` flipped to `true` on BOTH the
  Velero server and the Restic node-agent ServiceAccounts. Both
  controllers MUST call the Kubernetes API (Velero for cross-namespace
  resource enumeration / Backup-Restore CRDs; Restic for
  PodVolumeBackup / PodVolumeRestore / ResticRepository / coordination
  leases). Disabling token automount without a projected-token
  workaround silently broke the controllers.
* Removed `helm/templates/velero-secret.yaml`. The chart was rendering
  a placeholder Secret named `cluster-backup-cloud-credentials` which
  collided with the real Secret that `deploy.sh` creates out-of-band
  (`helm upgrade` would fail with "resource already exists" on every
  re-install). The Secret is now solely owned by `deploy.sh`.
  Removed the corresponding `checksum/secret` annotation from the
  Velero Deployment and Restic DaemonSet.
* Added `helm/templates/velero-service.yaml`. The chart previously
  shipped a `ServiceMonitor` selecting a Service that did not exist,
  and `NOTES.txt` instructed users to
  `kubectl port-forward svc/<fullname>-velero` — also missing.
  Service is `ClusterIP`, metrics-only.
* `helm/templates/networkpolicy.yaml` egress no longer uses `to: []`
  for any rule. DNS is constrained to the kube-system namespace;
  external S3 traffic is allowed to `0.0.0.0/0` MINUS RFC1918,
  link-local, and loopback ranges (so restic / velero can never
  accidentally exfiltrate to in-cluster IPs over an S3-shaped path);
  Kubernetes API access is constrained to the `default` namespace
  ClusterIP.
* `helm/templates/restic-daemonset.yaml` no longer mounts host `/` via
  `hostPath`. Restic only needs read access to `/var/lib/kubelet/pods`
  (the standard kubelet pod-volume root), so that's the only host
  mount that remains. Mounting host-root was incompatible with Pod
  Security `restricted` and not required for restic operation.
* `helm/templates/backup-schedules.yaml` passes a full
  `dict "Chart" $.Chart "Release" $.Release "Values" $.Values
   "Template" $.Template ...` context into the schedule-naming and
  schedule-config helpers, removing any ambiguity about what
  `cluster-backup.fullname` and `cluster-backup.labels` resolve to
  when called from within a `range` loop body.
* `helm/templates/_helpers.tpl`: simplified
  `cluster-backup.serviceAccountName` and
  `cluster-backup.resticServiceAccountName` to return deterministic
  names. The previous form gated on `.Values.velero.{server,restic}
  .serviceAccount.create`, which is not a path that exists in
  `values.yaml` — it was a remnant of the now-removed `velero`
  subchart pattern. The helpers always returned `"default"`, which
  conflicted with the ClusterRoleBinding subject the chart actually
  installed; Velero would have been denied every API call.
* `helm/templates/NOTES.txt` rewritten: `kubectl create backup …` /
  `kubectl create restore …` are NOT valid kubectl subcommands for
  Velero CRDs and were misleading users into thinking the controller
  was broken when in reality they were running a non-existent
  kubectl verb. NOTES now uses the Velero CLI (`velero backup get`,
  `velero schedule get`, etc.) and points readers at the install
  docs. Same fix applied across README.md, deploy.sh, verify.sh,
  test-local.sh, and (post-deployment) usage messages.

Script hardening
----------------
* `deploy.sh`:
  - Switched to `#!/usr/bin/env bash` + `set -euo pipefail`.
  - All temporary files now live in a single `mktemp -d`-created
    directory with mode 0700, cleaned up via an `EXIT|INT|TERM` trap.
    The previous version wrote `/tmp/s3-credentials` and
    `/tmp/cluster-backup-values.yaml` at fixed paths — world-readable
    on some hosts and racy under concurrent runs.
  - S3 secret-key prompt uses `read -rs` (no echo). The credential is
    piped into `kubectl create secret … --from-file=cloud=/dev/stdin
    --dry-run=client -o yaml | kubectl apply -f -`, so the secret
    never lives on disk and never appears in argv (`ps`).
  - All configuration accepts env-var inputs (`TENANT`, `CLUSTER`,
    `ENVIRONMENT`, `S3_*`) so the script is usable in CI/automation
    without an interactive TTY.
* `test-local.sh`:
  - MinIO image pinned (`minio/minio:RELEASE.2024-08-17T01-24-54Z`).
    Was `minio/minio:latest`, which made local test runs
    non-reproducible and exposed them to silent upstream breaking
    changes.
  - MinIO credentials no longer hardcoded as `minioadmin / minioadmin`
    in the manifest. Access key defaults to `localdev-access`; secret
    key is a freshly-generated 24-char random value per run
    (overridable via `TEST_MINIO_{ACCESS,SECRET}_KEY` env vars).
    Rendered into the Secret over a stdin pipe — same no-argv-leak
    pattern as deploy.sh.
  - Service now correctly exposes BOTH the S3 API (NodePort 30000)
    AND the web console (NodePort 30001). The original Service mapped
    only 9000 to NodePort 30000 yet the "MinIO Console" line pointed
    at 30000 — that's the S3 API, not the console. Web console
    address is now `http://localhost:30001`.
  - Replaced fixed `sleep 30` waits with `wait_for_velero_phase` which
    polls `.status.phase` of the Backup / Restore CR and emits
    `kubectl describe` + `velero backup logs` diagnostics on timeout.
  - Uses the Velero CLI for backup / restore creation.
* `verify.sh`:
  - Same shebang + strict-mode + Velero-CLI updates as deploy.sh.
  - `test_backup_creation` polls phase with diagnostics instead of
    blind-sleeping; also skips gracefully if the Velero CLI isn't
    installed locally (verify is run on hosts that may not have it).
* `create-tenant-cluster.sh`:
  - All upstream components are version-pinned via env-var defaults:
    `DOKS_K8S_VERSION`, `INGRESS_NGINX_VERSION`, `CERT_MANAGER_VERSION`,
    `METRICS_SERVER_VERSION`. The previous version used
    `doctl kubernetes cluster create … --version latest` and
    `metrics-server/releases/latest/`, making provisioning silently
    non-reproducible.
  - Team-member namespace is normalized to RFC1123
    (`AnnaF` → `annaf`, etc.) before being passed to `kubectl create
    namespace`. K8s rejects uppercase-containing namespace names with
    an opaque error, and the previous version of this script would
    silently fail at `create_team_member_namespace` for any contributor
    whose handle wasn't already lowercase-alphanumeric.
  - `create_team_member_access` no longer reads `.secrets[0].name` to
    harvest a long-lived ServiceAccount token (the BoundServiceAccount-
    Token feature in K8s ≥1.24 means that field is empty on modern
    clusters, so the function was returning an empty kubeconfig).
    Switched to `kubectl create token` (TokenRequest API) with an
    explicit 24h duration. Kubeconfig is written with `umask 077` so
    the file containing the bearer token is mode 0600.

Not in scope for this commit
----------------------------
* The Restic DaemonSet still ships with a `SYS_ADMIN` capability add
  and a non-root securityContext (uid 65534). Real-world restic
  operation typically requires either root or a relaxed
  podSecurityContext to read pod volumes owned by arbitrary uids;
  tightening this further requires a per-environment decision about
  the Pod Security Standard for the `velero` namespace. Flagged in
  `cluster-backup/CHANGELOG.md` under "Notes for operators".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… DO droplets (#27)

* feat(sandbox-docker): add copier template for AIO Sandbox on DO droplets

Adds a new copier project for deploying agent-infra/sandbox (browser, shell,
filesystem, VSCode, Jupyter, MCP servers in one container) on a DigitalOcean
droplet. Mirrors the keycloak-docker / anythingllm-docker pattern: Caddy
reverse proxy, Infisical runtime secret injection, OpenTofu droplet +
firewall + reserved IP, skinny volume backups with GFS retention.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Auto-PR: Merge branch 'feature/wordpress-docker-copier-template' into feature/wp-hardening-shd (#19)

* feat: codify PHP limits and block .user.ini per Task D152 & #264

* chore(helm): implement multi-site values strategy for burnedout and ptoken

* chore: bump helm chart version to 3.2.7

* fix: resolve trivy security scan and linting issues

* udated version bump

* fix: resolve yamllint line endings and comment indentation

* fix: resolve yamllint line endings

* docs: apply markdownlint autofixes for PR #15 CI

- ADR-004: add blank line before bulleted list (MD032)
- workflows/README.md: switch *emphasis* to _emphasis_ for style consistency (MD049)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(wordpress/helm): address PR #15 review items 10-12

- ingress.yaml: parameterize spec.tls[0].secretName from
  .Values.ingress.tls[0].secretName so per-site overrides
  (burnedout-tls, ptoken-tls) actually take effect. Falls
  back to "wordpress-tls" when .Values.ingress.tls is a
  map or unset (preserves default and TLS hardening map).
- php-config-configmap.yaml: gate "auto_prepend_file =
  wordfence-waf.php" behind .Values.wordpress.wordfence.enabled
  (default false) to avoid PHP warnings when the plugin is
  not installed.
- ingress.yaml: gate the file-blocking server-snippet annotation
  behind .Values.ingress.serverSnippet.enabled (default true)
  so the chart can deploy on hardened controllers that set
  allow-snippet-annotations: false.
- values.yaml: add wordpress.wordfence.enabled and
  ingress.serverSnippet.enabled.
- Chart.yaml: bump 3.2.7 -> 3.3.0 (SemVer + WeOwnVer valid).
- wordpress/CHANGELOG.md: document 3.3.0.

Verified: helm template renders correctly with default,
values-burnedout, and values-ptoken; helm lint clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingress): expand edge block list and add security response headers

Addresses PR #19 review items 11 and 12:
- Block xmlrpc.php, wp-config backups, readme.html, license.txt,
  debug.log, and .svn at the nginx edge (defense-in-depth)
- Add X-Frame-Options, X-Content-Type-Options, HSTS, Referrer-Policy,
  Permissions-Policy, and Content-Security-Policy via configuration-snippet
- CSP is configurable per site via ingress.securityHeaders.contentSecurityPolicy
- Document wp-login.php rate limiting as controller-level config

Chart version 3.3.0 → 3.3.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(values): resolve dead config, env var duplicates, and size mismatch

- Move APACHE_HTTP_PORT from dead wordpress.extraEnvVars to top-level
  extraEnvVars (deployment.yaml only reads the top-level key)
- Remove duplicate WORDPRESS_ENABLE_REDIS (already injected by template
  when redis.enabled=true) and WORDPRESS_CONFIG_EXTRA (already generated
  from wordpressExtraWpConfigContent — duplicate silently overwrote the
  more complete template version)
- Align proxy-body-size 64m -> 128m to match PHP upload_max_filesize
- Fix comment: global.domain -> wordpress.domain
- Remove dead hosts field from per-site tls overrides (template builds
  hosts from wordpress.domain, not tls list)
- Document enableMultisite/redirectFromWWW as not yet wired

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: m.shahid <m.shahid538@yahoo.com>
Co-authored-by: Nik <nik.cimino@gmail.com>
Co-authored-by: romandidomizio <rodi1364@colorado.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* - SearXNG Deployment to 'searxng.weown.app'.

- Add SearXNG deployment and team integration docs

- Adds the production deployment assets, ingress routing, browser integration guide, workstation browser policy playbook, and secret-safe SearXNG settings templating so the service can be reviewed without committing production secrets.

* refactor(searxng): restructure as copier template matching keycloak-docker pattern

Resolves all 15 review comments from PR #20 code review:

BLOCKING (4 resolved):
- Remove committed production IPs from inventory.ini (no inventory file;
  copier template uses dynamic doctl-based discovery)
- Remove committed RFC1918 IPs from k8s-external-ingress.yaml (k8s
  manifests removed; Caddy handles TLS directly on droplet)
- Fix TLS bypass: old docker-compose bound port 80 publicly; now Caddy
  handles all TLS termination with auto Let's Encrypt
- Pin container images: configurable image variables replace :latest tags

SECURITY (4 resolved):
- Secret key no longer world-readable on disk: Infisical Machine Identity
  injects SEARXNG_SECRET_KEY at runtime via `infisical run`
- Disable Google autocomplete (privacy leak): set autocomplete to ""
- TLS cipher restrictions: Caddy enforces modern TLS by default (no nginx
  ingress annotations needed)
- Remove StrictHostKeyChecking=accept-new: no inventory.ini

OPERATIONAL (5 resolved):
- Docker installation via get.docker.com (upstream docker-ce, not
  mismatched docker.io + docker-compose-plugin)
- Ansible deploy uses proper changed_when patterns
- Health checks defined for all 3 services (searxng, valkey, caddy)
- Valkey actually configured: settings.yml includes redis.url pointing
  to valkey:6379 for rate-limiting and bot detection
- Firefox policies.json: slurp-merge-write pattern preserves existing
  enterprise policies instead of overwriting

TEMPLATE STRUCTURE (new):
- copier.yaml with _subdirectory: template (copier >= 9.0.0)
- template/terraform/ — main.tf (droplet + reserved IP + firewall),
  monitoring.tf (CPU/mem/disk alerts), variables.tf, outputs.tf,
  backend.tf, versions.tf, cloud-init.yaml
- template/docker/ — compose.prod.yaml, Caddyfile, searxng/settings.yml
- template/scripts/ — deploy.sh, backup.sh (skinny backups with DO
  Spaces offload), restore.sh
- template/ansible/ — deploy.yml, configure-browser-search.yml
- template/ — .gitignore, README.md, CHANGELOG.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(wordpress-docker): use Minimus private image with PDO extensions (D117)

Replace reg.mini.dev/wordpress:latest with
reg.mini.dev/1923/wordpress-fluentsmtp:latest in the copier template
default and all generated site configs.

The base Minimus WordPress image strips PDO (WordPress core uses
mysqli). FluentSMTP requires PDO for email logging — without it,
logins trigger a fatal error. The private image includes php-pdo-auto
and php-pdo-mysql-auto plus 5 other PHP extensions.

Ref: burnedout.xyz PDO postmortem 2026-04-14, decision D117.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ci): align drifted chart versions; tighten CHANGELOG consistency check

The version-consistency check in .github/workflows/validation.yml used
`grep -qi $chart_version $changelog_file` (presence-anywhere), which
silently passed even when Chart.yaml drifted away from the latest
CHANGELOG entry. Concrete case discovered during PR #19 review:
wordpress/helm/Chart.yaml was 3.2.6 while wordpress/CHANGELOG.md's top
entry was [3.3.8] — the older 3.2.6 string still appeared in past
entries, so the check kept passing.

Tighten the check to compare Chart.yaml's `version:` against the first
`## [X.Y.Z]` header in CHANGELOG.md (the Keep-a-Changelog newest-on-top
convention).

Three charts had pre-existing drift and would fail the new check on
main, so align them in this commit:

  nextcloud/helm/Chart.yaml      1.0.0 → 1.1.0
  vaultwarden/helm/Chart.yaml    1.0.0 → 1.3.1
  wordpress/helm/Chart.yaml      3.2.6 → 3.3.8

These are metadata-only bumps: Chart.yaml now matches the version each
chart's own CHANGELOG already declares as latest. No template or values
changes.

Not addressed in this commit (separate decisions needed):

- WeOwnVer format inconsistency between docs/VERSIONING_WEOWNVER.md
  (SEASON.MONTH.WEEK.ITERATION) and validation.yml's `versioning` job
  (SEASON.WEEK.DAY.VERSION). None of the current chart versions cleanly
  fit either schema.
- The `versioning` job only validates the first Chart.yaml found
  alphabetically (`*/helm/Chart.yaml | head -1` on line 212), not all
  of them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): split long error message to resolve yamllint line-length warning

* DR initial commit with restic + velero

* fix(cluster-backup): address all reviewer + CI feedback from PR #14/#24

Resolves the 3 failing CI jobs and all 30 inline review comments on
PR #24 (and the predecessor PR #14, whose comments were ported via the
branch rename `dra` → `feature/shahid-velero-restic`).

CI failures fixed
-----------------
* Helm Template Validation — chart no longer declares an upstream
  `velero` subchart dependency (the chart provides its OWN Velero +
  Restic templates; the two would collide and `helm template` could
  not run because `helm dependency build` is not invoked in CI).
* Helm Lint — removed the duplicate `cluster-backup.restoreName`
  template definition in `_helpers.tpl`.
* Documentation Validation / Markdown Lint — `cluster-backup/README.md`
  now satisfies MD031/MD022/MD032 (blank lines around fences /
  headings / lists). Added `cluster-backup/CHANGELOG.md` with a
  `[1.0.0] - 2026-05-22` entry that matches `Chart.yaml`'s
  `version: 1.0.0` — required by the tightened version-consistency
  check landed in #21.
* trailing-whitespace / end-of-file-fixer — stripped trailing whitespace
  from every cluster-backup yaml + shell file and ensured every file
  ends with `\n`.

Security & correctness (per Copilot / @romandidomizio review)
------------------------------------------------------------
* `automountServiceAccountToken: false` flipped to `true` on BOTH the
  Velero server and the Restic node-agent ServiceAccounts. Both
  controllers MUST call the Kubernetes API (Velero for cross-namespace
  resource enumeration / Backup-Restore CRDs; Restic for
  PodVolumeBackup / PodVolumeRestore / ResticRepository / coordination
  leases). Disabling token automount without a projected-token
  workaround silently broke the controllers.
* Removed `helm/templates/velero-secret.yaml`. The chart was rendering
  a placeholder Secret named `cluster-backup-cloud-credentials` which
  collided with the real Secret that `deploy.sh` creates out-of-band
  (`helm upgrade` would fail with "resource already exists" on every
  re-install). The Secret is now solely owned by `deploy.sh`.
  Removed the corresponding `checksum/secret` annotation from the
  Velero Deployment and Restic DaemonSet.
* Added `helm/templates/velero-service.yaml`. The chart previously
  shipped a `ServiceMonitor` selecting a Service that did not exist,
  and `NOTES.txt` instructed users to
  `kubectl port-forward svc/<fullname>-velero` — also missing.
  Service is `ClusterIP`, metrics-only.
* `helm/templates/networkpolicy.yaml` egress no longer uses `to: []`
  for any rule. DNS is constrained to the kube-system namespace;
  external S3 traffic is allowed to `0.0.0.0/0` MINUS RFC1918,
  link-local, and loopback ranges (so restic / velero can never
  accidentally exfiltrate to in-cluster IPs over an S3-shaped path);
  Kubernetes API access is constrained to the `default` namespace
  ClusterIP.
* `helm/templates/restic-daemonset.yaml` no longer mounts host `/` via
  `hostPath`. Restic only needs read access to `/var/lib/kubelet/pods`
  (the standard kubelet pod-volume root), so that's the only host
  mount that remains. Mounting host-root was incompatible with Pod
  Security `restricted` and not required for restic operation.
* `helm/templates/backup-schedules.yaml` passes a full
  `dict "Chart" $.Chart "Release" $.Release "Values" $.Values
   "Template" $.Template ...` context into the schedule-naming and
  schedule-config helpers, removing any ambiguity about what
  `cluster-backup.fullname` and `cluster-backup.labels` resolve to
  when called from within a `range` loop body.
* `helm/templates/_helpers.tpl`: simplified
  `cluster-backup.serviceAccountName` and
  `cluster-backup.resticServiceAccountName` to return deterministic
  names. The previous form gated on `.Values.velero.{server,restic}
  .serviceAccount.create`, which is not a path that exists in
  `values.yaml` — it was a remnant of the now-removed `velero`
  subchart pattern. The helpers always returned `"default"`, which
  conflicted with the ClusterRoleBinding subject the chart actually
  installed; Velero would have been denied every API call.
* `helm/templates/NOTES.txt` rewritten: `kubectl create backup …` /
  `kubectl create restore …` are NOT valid kubectl subcommands for
  Velero CRDs and were misleading users into thinking the controller
  was broken when in reality they were running a non-existent
  kubectl verb. NOTES now uses the Velero CLI (`velero backup get`,
  `velero schedule get`, etc.) and points readers at the install
  docs. Same fix applied across README.md, deploy.sh, verify.sh,
  test-local.sh, and (post-deployment) usage messages.

Script hardening
----------------
* `deploy.sh`:
  - Switched to `#!/usr/bin/env bash` + `set -euo pipefail`.
  - All temporary files now live in a single `mktemp -d`-created
    directory with mode 0700, cleaned up via an `EXIT|INT|TERM` trap.
    The previous version wrote `/tmp/s3-credentials` and
    `/tmp/cluster-backup-values.yaml` at fixed paths — world-readable
    on some hosts and racy under concurrent runs.
  - S3 secret-key prompt uses `read -rs` (no echo). The credential is
    piped into `kubectl create secret … --from-file=cloud=/dev/stdin
    --dry-run=client -o yaml | kubectl apply -f -`, so the secret
    never lives on disk and never appears in argv (`ps`).
  - All configuration accepts env-var inputs (`TENANT`, `CLUSTER`,
    `ENVIRONMENT`, `S3_*`) so the script is usable in CI/automation
    without an interactive TTY.
* `test-local.sh`:
  - MinIO image pinned (`minio/minio:RELEASE.2024-08-17T01-24-54Z`).
    Was `minio/minio:latest`, which made local test runs
    non-reproducible and exposed them to silent upstream breaking
    changes.
  - MinIO credentials no longer hardcoded as `minioadmin / minioadmin`
    in the manifest. Access key defaults to `localdev-access`; secret
    key is a freshly-generated 24-char random value per run
    (overridable via `TEST_MINIO_{ACCESS,SECRET}_KEY` env vars).
    Rendered into the Secret over a stdin pipe — same no-argv-leak
    pattern as deploy.sh.
  - Service now correctly exposes BOTH the S3 API (NodePort 30000)
    AND the web console (NodePort 30001). The original Service mapped
    only 9000 to NodePort 30000 yet the "MinIO Console" line pointed
    at 30000 — that's the S3 API, not the console. Web console
    address is now `http://localhost:30001`.
  - Replaced fixed `sleep 30` waits with `wait_for_velero_phase` which
    polls `.status.phase` of the Backup / Restore CR and emits
    `kubectl describe` + `velero backup logs` diagnostics on timeout.
  - Uses the Velero CLI for backup / restore creation.
* `verify.sh`:
  - Same shebang + strict-mode + Velero-CLI updates as deploy.sh.
  - `test_backup_creation` polls phase with diagnostics instead of
    blind-sleeping; also skips gracefully if the Velero CLI isn't
    installed locally (verify is run on hosts that may not have it).
* `create-tenant-cluster.sh`:
  - All upstream components are version-pinned via env-var defaults:
    `DOKS_K8S_VERSION`, `INGRESS_NGINX_VERSION`, `CERT_MANAGER_VERSION`,
    `METRICS_SERVER_VERSION`. The previous version used
    `doctl kubernetes cluster create … --version latest` and
    `metrics-server/releases/latest/`, making provisioning silently
    non-reproducible.
  - Team-member namespace is normalized to RFC1123
    (`AnnaF` → `annaf`, etc.) before being passed to `kubectl create
    namespace`. K8s rejects uppercase-containing namespace names with
    an opaque error, and the previous version of this script would
    silently fail at `create_team_member_namespace` for any contributor
    whose handle wasn't already lowercase-alphanumeric.
  - `create_team_member_access` no longer reads `.secrets[0].name` to
    harvest a long-lived ServiceAccount token (the BoundServiceAccount-
    Token feature in K8s ≥1.24 means that field is empty on modern
    clusters, so the function was returning an empty kubeconfig).
    Switched to `kubectl create token` (TokenRequest API) with an
    explicit 24h duration. Kubeconfig is written with `umask 077` so
    the file containing the bearer token is mode 0600.

Not in scope for this commit
----------------------------
* The Restic DaemonSet still ships with a `SYS_ADMIN` capability add
  and a non-root securityContext (uid 65534). Real-world restic
  operation typically requires either root or a relaxed
  podSecurityContext to read pod volumes owned by arbitrary uids;
  tightening this further requires a per-environment decision about
  the Pod Security Standard for the `velero` namespace. Flagged in
  `cluster-backup/CHANGELOG.md` under "Notes for operators".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cluster-backup): repair Schedule rendering + dedupe component label

Two issues that surfaced once kubeconform actually parsed the rendered
output (the previous CI run was already past my first commit but failed
on Kubeconform Validation):

1. Duplicate `app.kubernetes.io/component` label

   The common `cluster-backup.labels` helper emitted
   `app.kubernetes.io/component: backup-system`, and every per-resource
   template also appended its own `app.kubernetes.io/component: <role>`
   right below the include. The K8s API server tolerates duplicate map
   keys (last write wins), but strict YAML parsers — including
   `kubeconform` — reject the manifest. Removed the component-label
   line from the shared helper; per-resource components remain.

2. Schedule CR `.spec.schedule` rendered as `"map[…]"` instead of cron

   The `cluster-backup.backupScheduleConfig` helper accessed
   `.schedule | quote` at the top level, but its caller passes the
   whole schedule object under `.schedule` — so `.schedule` was the
   FULL map, and Go's `fmt.Sprint` rendered it as
   `"map[enabled:true excludeNamespaces:[…] includeNamespaces:[]
   retention:30d schedule:0 2 * * *]"`. Velero would have rejected
   the Schedule as invalid syntax at apply time.

   Reworked the helper to extract fields explicitly via
   `.schedule.schedule`, `.schedule.retention`,
   `.schedule.includeNamespaces`, `.schedule.excludeNamespaces`.
   This is the same access pattern the caller already implied; the
   helper had just been written incorrectly from the start. The CR
   now renders with the expected `schedule: "0 2 * * *"`.

   Also fixed an adjacent issue in `backup-schedules.yaml`: the
   `{{- $ctx := dict … -}}` line had its trailing newline stripped,
   which fused the previous Schedule's last YAML line directly to
   the next `---` document separator (e.g. `defaultVolumesToRestic:
   true---`). Replaced with non-stripped `{{ $ctx := merge … }}` on
   its own line.

Verified with `helm template … | kubeconform -strict -summary
-ignore-missing-schemas -`:
  21 resources, Valid: 11, Invalid: 0, Errors: 0, Skipped: 10
(skipped resources are Velero CRDs with no published JSON schema:
 BackupStorageLocation, VolumeSnapshotLocation, Schedule).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cluster-backup): resolve Trivy CRITICAL/HIGH findings

Local `trivy config --severity CRITICAL,HIGH cluster-backup/` is now
clean (0 findings). Two categories:

Real code fixes
---------------
1. **KSV-0053 (HIGH) — restic ClusterRole granted `pods/exec`.**
   Restic backs up pod volumes by reading the kubelet bind-mount; it
   never execs into user pods. Removed `pods/exec` from the resource
   list and tightened the verb set on `[pods, pods/log, namespaces,
   nodes]` from `get list watch create update patch delete` down to
   just `get list watch` — restic does not own any of those resources.
   This also drops KSV-0042 (delete pod logs) + part of KSV-0048
   (manage workloads/pods) from the restic role specifically.

2. **KSV-0005 (HIGH) — restic securityContext was internally
   inconsistent.** The previous values.yaml had
   `runAsNonRoot: true, runAsUser: 65534` + `capabilities.add:
   [SYS_ADMIN]`. uid 65534 cannot read pod volumes owned by arbitrary
   uids, so restic would have failed at runtime. Per Velero's
   documented restic deployment posture (https://velero.io/docs/v1.12/restic/),
   the node-agent must run as root with SYS_ADMIN. Changed to
   `runAsNonRoot: false, runAsUser: 0, runAsGroup: 0, fsGroup: 0`,
   keeping `readOnlyRootFilesystem: true` and the default seccomp
   profile. The PSS for the `velero` namespace must therefore be
   `baseline` or `privileged` (NOT `restricted`) — documented in
   cluster-backup/CHANGELOG.md.

Targeted .trivyignore additions
-------------------------------
The remaining findings are inherent to a cluster-backup tool and
cannot be fixed without breaking its function. Each entry in
.trivyignore is explained — Trivy ignore-rules are the right vehicle
for "this is intentional, here's why":

- KSV-0041 (CRITICAL): Velero must manage Secrets to back them up.
- KSV-0056 (HIGH x3): Velero must manage Services/Endpoints/Ingresses/
  NetworkPolicies to back up and restore network topology.
- KSV-0005 / KSV-0022 (HIGH): SYS_ADMIN is required for restic's
  mount/setns ops; this is Velero's documented deployment posture.
- KSV-0012 (MEDIUM): companion of KSV-0005 (run-as-root).
- KSV-0023 (MEDIUM): hostPath on `/var/lib/kubelet/pods` is how restic
  reads pod volumes. The earlier broader mount of host `/` was already
  removed in d0c637b.
- KSV-0042 (MEDIUM): Velero must delete pods to restore them.
- KSV-0048 (MEDIUM): Velero must create/update Deployments,
  StatefulSets, DaemonSets, Jobs, CronJobs.
- KSV-0049 (MEDIUM): Velero must manage ConfigMaps.
- KSV-0125 (MEDIUM): `velero/velero:v1.12.2` is the official upstream
  image on docker.io. Mirroring to a private registry is tracked as
  a separate follow-up.

All ignored rules are scoped by .trivyignore comments to the
cluster-backup chart only.

Roman's review verified
-----------------------
All 17 items from @romandidomizio's CHANGES_REQUESTED review have been
addressed in this branch (d0c637b + ac2e9f6 + this commit). Spot-checked
all of them; no outstanding source-level issues.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(wordpress-docker): replace .env secrets with Infisical runtime injection

Removes all on-disk .env.prod secret management and replaces it with
the Infisical Universal Auth (Machine Identity) pattern, matching the
keycloak-docker reference implementation.

Changes:
- copier.yaml: enable_infisical defaults true; prompts for client_id
  and client_secret instead of legacy infisical_token
- variables.tf.jinja: replace infisical_token + enable_infisical bool
  with infisical_client_id + infisical_client_secret (sensitive);
  mysql_password / mysql_root_password gated on not enable_infisical
- main.tf.jinja: templatefile() passes correct vars per enable_infisical
- cloud-init.yaml.jinja: installs Infisical CLI, writes infisical-auth.sh
  (machine identity), starts stack via infisical run, cron backup
  authenticated via infisical-auth.sh + infisical run
- terraform.tfvars.example.jinja: removes mysql passwords from Required
  section; replaces infisical_token with client_id + client_secret
- ansible/deploy.yml.jinja: removes pre_tasks (.env.prod check), removes
  Upload .env task, converts docker_compose_v2 calls to shell + infisical run
- scripts/deploy.sh.jinja: removes .env.prod check + scp; uses infisical run
- scripts/backup.sh.jinja: removes source .env; uses infisical run for DB dump
- scripts/restore.sh.jinja: removes source .env + dot-env restore; uses
  infisical run for DB restore and compose up
- docker/.env.prod.example: replaced with Infisical pointer (no passwords)

Compliance: §3.10 Infisical checklist, §3.1 NIST PR.DS, §3.2 CIS 3.11,
            §3.4 ISO A.8.24, §3.5 SOC2 CC6.7, §3.8 Docker Compose checklist

* feat(wordpress-docker): add pull-prod workflow + fix backup/restore scripts

- Add scripts/pull-prod.sh: pulls production DB + wp-content to local
  dev stack in one command; reads DB password from docker inspect (not
  .env) to avoid silent dump failures when credentials diverge
- Fix backup.sh: use docker inspect for DB password instead of sourcing
  .env — same root cause that caused 0-byte dumps in burnedout-xyz
- Fix restore.sh: add COMPOSE_PROJECT_NAME=burnedout-local, auto URL
  replacement (siteurl/home) after prod DB import, use mariadb client
- Fix compose.local.yaml: add Caddy service (WP image is FPM-only;
  missing Caddy made the stack silently unreachable on port 8080)
- Add Caddyfile.local: HTTP-only local Caddy config for FPM proxy
- Propagate all fixes to template (pull-prod.sh.jinja, backup.sh.jinja,
  restore.sh.jinja, compose.local.yaml.jinja)
- Expand README.md (site + template) with Day-to-Day Operations section
  covering when/how to use each script

Fixes: silent 0-byte mysqldump when .env password diverges from
running container; localhost:8080 returning empty due to missing Caddy

* chore: add Ansible template and documentation files

- RESTORE-RUNBOOK-PROMPT.md: runbook documentation for burnedout.xyz restore
- scripts/manage-droplets.sh: droplet management script (fixed SC2155 shellcheck)
- wordpress-docker/template/ansible/: Ansible configuration, inventory template, and requirements

* fix(wordpress-docker): address code review findings

Critical fixes:
- restore.sh: fix PROJECT_NAME ("burnedout" not "burnedoutxyz") and
  APP_DIR ("/opt/burnedout" not "/opt/burnedout-xyz") — remote restore
  was completely broken, targeting nonexistent containers and paths
- restore.sh: use docker inspect for DB credentials instead of sourcing
  .env — matches backup.sh pattern, works with or without Infisical
- RESTORE-RUNBOOK: replace broken base image (reg.mini.dev/wordpress:latest)
  with correct private image (reg.mini.dev/1923/wordpress-fluentsmtp:latest)

Medium fixes:
- backup.sh: stop including .env in backup archive — production secrets
  should not be stored in tarballs (credential exposure vector)
- RESTORE-RUNBOOK: fix false claim that compose.local.yaml sets
  WORDPRESS_CONFIG_EXTRA (only compose.prod.yaml does)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Auto-PR: feat: add observability infrastructure, SearXNG copier template, and fleet tooling (#26)

* feat: add observability infrastructure, SearXNG copier template, and fleet tooling

Add SigNoz self-hosted observability (signoz-docker/) and OTel Collector
agent (otel-agent/) for per-container metrics, log aggregation, and
traces across the fleet. Add SearXNG copier template (searxng-docker/)
with full IaC, Infisical integration, skinny backups, and browser search
playbook. Add fleet scripts for DO agent enablement and OTel deployment.
Add AnythingLLM MCP configuration playbook. Fix cloud-init `runcmds:`
typo to correct `runcmd:` across all templates. Fix manage-droplets.sh
SSH key rotation parsing bug. Add project CLAUDE.md for AI guidance.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(otel): SigNoz Cloud fleet agent Phase 1 (burnedout-xyz verified)
Pivot observability to SigNoz Cloud us2 with per-droplet OTel collector,
Infisical runtime secret injection, bootstrap/deploy fleet scripts, and
deprecation notes for self-hosted signoz-docker template.

* fix(pr26): address Copilot + reviewer findings on observability work

Resolves all blocking review feedback on PR #26 surfaced by Copilot
(rounds 1 + 2) and an independent code review of the diff. Covers
correctness bugs, security defaults, and operational gaps. Scope is
limited to closing review items — no new features.

Correctness bugs (deploy-blocking):
- signoz scripts (deploy/backup/restore + ansible/deploy.yml.jinja): use
  `replace('-', '_')` to match `main.tf.jinja` -> `templatefile` -> cloud-init
  path/volume normalization. Previously the no-separator form (`replace('-', '')`)
  caused /opt/signozobservability vs /opt/signoz_observability divergence and
  silent volume-name mismatches in backup/restore.
- cloud-init OTel gateway DSN: switch `password=$${CLICKHOUSE_PASSWORD}` to
  `password=$${env:CLICKHOUSE_PASSWORD}` so the OTel collector actually expands
  the env var (bare `${VAR}` is not interpolated by the collector).
- README.md.jinja: fix broken code fence at line 92 ("```nished Phase 1...").
- README.md.jinja: replace undefined `monitoring_*_threshold` references with
  the actual copier-defined names (`cpu_alert_threshold`, etc.).
- monitoring.tf.jinja: hardcoded `value = 8` for load-5 alert is now driven
  by a new `load_alert_threshold` copier variable (default 8 = 2x default vCPUs).
- CHANGELOG.md.jinja: drop the stub `[{{ version }}] - {{ date }}` line that
  referenced undefined copier variables.
- anythingllm/ansible/configure-allm.yml: rewrite to use plain Ansible
  templating (`{{ var }}`) instead of broken copier-style escapes
  (`{{ '{{' }} var {{ '}}' }}`). Wrap Docker Go-template (`{{.State.Running}}`)
  in `{% raw %}...{% endraw %}` so Ansible passes it through. Add
  `no_log: true` to tasks that handle merged MCP server configs (which can
  include credentials from `extra_mcp_servers`).
- otel-agent OTLP transport docs: fix gRPC/HTTP mismatch in comments across
  config.yaml, compose.yaml, deploy-otel-fleet.sh, otel-agent/README.md
  (the actual exporter is otlphttp).
- scripts/enable-do-agent.sh: drop unused `ALREADY=0` counter.

Security + IaC fixes:
- terraform backend.tf (signoz + searxng): remove invalid `var.*` references
  in the s3 backend block (backend cannot interpolate variables — init runs
  before vars are evaluated). Add the working DO Spaces pattern from
  keycloak-docker/sites/sso.weown.dev/ (skip_* flags) + a new `init.sh.jinja`
  that bridges terraform.tfvars -> `tofu init -backend-config=`. Also drop
  the unsupported `lock`/`lock_timeout` block. Moves `spaces_*` variable
  declarations to variables.tf.jinja and adds placeholders to
  terraform.tfvars.example.jinja.
- New `ssh_source_cidrs` copier variable (signoz + searxng) drives the SSH
  firewall rule. Default keeps backward-compat `["0.0.0.0/0", "::/0"]` with
  a loud help warning to pin to admin IP/32 or VPN range in production.
- signoz cloud-init Infisical CLI install: switch from the deprecated
  install-cli.sh channel (capped at v0.38 — broken `infisical run`) to the
  current artifacts-cli.infisical.com apt repo, mirroring the idempotent
  block in scripts/bootstrap-otel-agent.sh.
- searxng copier.yaml: pin `searxng_image` to a dated release tag (was
  `:latest` — drift risk on every pull); help text directs operators to
  Docker Hub for newer tags.
- searxng .gitignore: rewrite the `searxng/settings.yml` ignore rule to the
  actual generated path `docker/searxng/settings.yml`.
- restore.sh.jinja: validate `BACKUP_NAME` against a strict allowlist regex
  `^[A-Za-z0-9._-]+$` before forwarding it into the remote ssh `bash -c`
  command, closing the shell-injection vector flagged by both reviewers.
- otel-agent deploy paths (deploy-otel-fleet.sh + otel-agent/deploy.yml):
  reject `http://` URLs explicitly in the OTEL_URL normalization case; only
  `https://` is accepted (config.yaml sets `insecure: false`).
- otel-agent/README.md: add a "Threat model — host-level access by design"
  section that explicitly documents the root+docker.sock+host-root-mount
  topology, what `:ro` mitigates vs doesn't, and the mitigations the design
  relies on.

Operational quality:
- copier.yaml (signoz + searxng): remove `s3` from the
  `backup_remote_storage` choices — the backup/restore scripts only
  implement `do-spaces`, so picking `s3` was a silent no-op.
- copier.yaml + variables.tf.jinja + README.md.jinja (signoz): reword
  `clickhouse_retention_days` description to mark it as informational
  (actual TTL is set via SigNoz UI), since the value was never wired into
  any ClickHouse config.
- compose.prod.yaml.jinja + cloud-init.yaml.jinja (signoz + searxng): add
  `/var/log/caddy:/var/log/caddy` bind mount so Caddy access logs survive
  container recreation and the otel-agent filelog/caddy receiver can pick
  them up from the host.

Repo hygiene:
- Add `.claude/` to top-level `.gitignore` (Claude Code local workspace).

Deferred (require design decisions; tracked for follow-up):
- Infisical Machine Identity persistence in user_data / TF state. Documented
  as known risk in CHANGELOG; full remediation requires architectural change
  (cloud-init bootcmd + tmpfs, systemd credentials, or external bootstrap
  delivery). Rotate the Machine Identity after destroy.
- `lifecycle { ignore_changes = [user_data] }` masks user_data updates
  (intentional, to avoid droplet recreation on every var bump). Future:
  document `tofu taint` workflow in signoz-docker README.
- Docker-socket-proxy in front of otel-agent (significant scope, follow-up PR).
- The legacy `install-cli.sh` Infisical install pattern in `anythingllm-docker/`
  and `keycloak-docker/` cloud-init.yaml.jinja (out of this PR's scope; same
  fix applies — separate PR).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pr26): address Copilot round-3 + own review pass

Resolves the 22 new Copilot comments on merge commit 59a3d0a plus
additional issues found in an own-review pass over the diff.

New Copilot findings (all addressed):
- signoz-docker/template/README.md.jinja: quick-start used `tofu init`
  directly, but the S3 backend requires `-backend-config` flags via the
  new `./init.sh`. Rewrote step 2 to use the init script + documented
  why (variables not allowed in backend blocks).
- signoz-docker/template/README.md.jinja: step 3 referenced the old
  positional-arg `deploy-otel-fleet.sh weown-ai <ip>` syntax which was
  removed in the SigNoz Cloud pivot. Rewrote to use the new flag-based
  selectors and added context that fleet OTel agents ship to SigNoz
  Cloud by default (not to this self-hosted gateway).
- signoz-docker/template/scripts/{deploy,restore}.sh.jinja: replaced
  `143.198.xxx.xxx` example IPs (real DO range) with RFC 5737 documentation
  ranges (`198.51.100.42`) so the public repo does not normalize references
  to real infrastructure.
- signoz-docker/template/terraform/templates/cloud-init.yaml.jinja:
  `docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'` would
  have been interpreted by Copier's Jinja pass as attribute lookups on the
  root context (likely failing rendering). Wrapped in `{% raw %}...{% endraw %}`.
- signoz-docker/template/terraform/templates/cloud-init.yaml.jinja:
  retention echo claimed "daily 30d / monthly 12mo / yearly forever" but
  the implementation only deletes local backups older than 30 days.
  Rewrote the message to match the actual behavior.
- scripts/bootstrap-otel-agent.sh: Infisical Machine Identity secrets were
  passed inline as env-var assignments in the ssh command string, which
  exposes them in the local `ps` listing during the SSH connection.
  Rewrote to pipe `export VAR=...` (via `printf %q` for safe quoting)
  through ssh stdin (process substitution) so the secrets never appear
  in argv.
- otel-agent/README.md: my prior "Threat model" section claimed that
  `:ro` on `/var/run/docker.sock` prevents Docker API writes. That is
  incorrect — `:ro` is a bind-mount flag affecting the socket inode,
  NOT the Docker API protocol. Any process that can `connect(2)` to the
  socket can issue write API calls (containers/create, exec, --privileged,
  etc.) regardless of mount mode. Rewrote the section to call out the
  socket access as effectively root-on-host, list what we actually rely
  on (image pinning, memory cap, no host-side secrets at rest), and
  document the docker-socket-proxy mitigation path for future work.
- otel-agent/compose.yaml + otel-agent/README.md "Safety" section:
  brought the "all mounts read-only ⇒ safe" claim in line with the
  corrected threat model — fs mounts are tamper-safe, but the docker
  socket isn't, and we say so explicitly.
- signoz-docker/template/docker/compose.prod.yaml.jinja +
  signoz-docker/template/terraform/templates/cloud-init.yaml.jinja:
  added a comment block on the ZooKeeper service explaining that
  `ALLOW_ANONYMOUS_LOGIN: "yes"` is intentional for this deprecated
  self-hosted fallback (port unpublished, only ClickHouse reaches it on
  `signoznet`); enabling SASL would require synchronized ClickHouse
  credentials and a new Infisical secret. Operators adopting this for
  production should enable auth before exposing the stack.

Own-review pass:
- scripts/deploy-otel-fleet.sh: updated the OTEL_URL normalization
  comment to match the actual behavior (https:// or scheme-less; reject
  plain http:// because `tls.insecure: false`). The previous comment
  said "otlphttp requires https:// or http://" which contradicted the
  reject-http logic.

Not addressed (deferred — same call as round 2):
- Infisical Machine Identity persisted in user_data / Terraform state
  (architectural change required; tracked in CHANGELOG).
- Copilot's two "configure-allm.yml jinja escapes" comments at line 33
  are stale — that file was fixed in a prior commit and currently uses
  plain `{{ var }}` form. No further action needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(anythingllm): move configure-allm.yml under anythingllm-docker/

The configure-allm.yml playbook uses `docker exec` + `docker inspect`,
so it is Docker-specific. The `anythingllm/` directory is reserved for
the helm/k8s deployment path; Docker-stack tooling belongs under
`anythingllm-docker/` next to the copier template + cloud-init that
provision the Docker-based instances this playbook configures.

Used `git mv` to preserve file history. The empty `anythingllm/ansible/`
directory has been removed. Updated the CHANGELOG entry to reflect the
new location and the rationale for the move.

No content changes — only relocation. Callers (currently none in-repo)
referencing `anythingllm/ansible/configure-allm.yml` should switch to
`anythingllm-docker/ansible/configure-allm.yml`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Nik <nik.cimino@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: m.shahid <m.shahid538@yahoo.com>
Co-authored-by: Nik Cimino <ncimino@users.noreply.github.com>

* Auto-PR: chore(s004): add baseline copier templates for monitoring and infrastructure scaffolding (#31)

* feat: add observability infrastructure, SearXNG copier template, and fleet tooling

Add SigNoz self-hosted observability (signoz-docker/) and OTel Collector
agent (otel-agent/) for per-container metrics, log aggregation, and
traces across the fleet. Add SearXNG copier template (searxng-docker/)
with full IaC, Infisical integration, skinny backups, and browser search
playbook. Add fleet scripts for DO agent enablement and OTel deployment.
Add AnythingLLM MCP configuration playbook. Fix cloud-init `runcmds:`
typo to correct `runcmd:` across all templates. Fix manage-droplets.sh
SSH key rotation parsing bug. Add project CLAUDE.md for AI guidance.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(s004): sync live caddy stdout configuration and private registry updates

* chore(s004): add baseline copier templates for monitoring and infrastructure scaffolding

* fix(s004-deployment): address Copilot + reviewer findings — make deploy actually work

Resolves 8 new Copilot comments on the merge commit + 12 findings from
an independent reviewer pass on s004-deployment/. The directory was
committed as a skeleton with several deployment-blocking bugs.

CRITICAL — deployment-blocking bugs:

- Volume name mismatch (backup.sh, restore.sh, cloud-init backup script):
  scripts referenced `${PROJECT_NAME}_storage` where `PROJECT_NAME=s004anythingllm`
  (no underscore), so the resolved name was `s004anythingllm_storage`. But
  `docker/compose.prod.yaml` (and cloud-init's embedded compose) explicitly
  name volumes `s004_anythingllm_storage`. Every backup tarred an empty
  anonymous volume; every restore wrote into a phantom volume the app never
  reads. Hardcoded the actual volume names (`s004_anythingllm_storage`,
  `s004_anythingllm_caddy_data`) in all three sites.
- Empty Infisical credentials in cloud-init: the `infisical-auth.sh` helper,
  daily cron wrapper, and runcmd `infisical run` all had literal
  `INFISICAL_CLIENT_ID=''` / `--projectId=`. terraform/main.tf already plumbs
  `infisical_client_id`, `infisical_client_secret`, `infisical_project_id`,
  `infisical_environment` into `templatefile()`, so just reference them as
  `${infisical_*}`.
- Empty `--projectId=` in scripts/deploy.sh + scripts/restore.sh: workstation
  scripts now read INFISICAL_PROJECT_ID (and optional INFISICAL_ENV, default
  `prod`) from the environment with a `:?` guard that prints a clear error.
- Hardcoded `s004.ccc.bot` domain in cloud-init compose env + Caddyfile:
  switched to `${domain}` so var.domain takes effect.
- Image registry inconsistency: cloud-init pulled `mintplexlabs/anythingllm:latest`,
  compose used `reg.mini.dev/anythingllm:latest`. Both now use
  `${anythingllm_image}` from var.anythingllm_image
  (default `reg.mini.dev/anythingllm:1.7.2`, the WeOwn mirror).
- 23 unescaped shell variables in cloud-init `templatefile()` body
  (`${JWT_SECRET}`, `${ADMIN_EMAIL}`, `${BACKUP_NAME}`, `${SPACES_BUCKET}`,
  `${BASH_REMATCH[1]}`, etc.) would have made `tofu plan` error before
  producing any output. Escaped every uppercase shell var as `$${VAR}`.

HIGH — lessons from PR #26 applied here:

- SSH 0.0.0.0/0 firewall rule → introduced `var.ssh_source_cidrs`
  (default `["0.0.0.0/0", "::/0"]` with help text directing production
  to pin to admin/VPN CIDR). Same fix as signoz-docker.
- Deprecated Infisical install channel (`infisical.com/install-cli.sh`,
  capped at v0.38 with broken `infisical run` session handling) →
  rewrote to use the current `artifacts-cli.infisical.com` apt repo with
  the same idempotent legacy-purge logic from `bootstrap-otel-agent.sh`.
- Caddy access logs written to `/var/log/caddy/` but no host bind mount
  → added `- /var/log/caddy:/var/log/caddy` to both `docker/compose.prod.yaml`
  and the cloud-init embedded compose, so otel-agent's filelog/caddy
  receiver can read them.
- `restore.sh` shell-injection via unvalidated `$BACKUP_NAME` →
  `[[ ! "$BACKUP_NAME" =~ ^[A-Za-z0-9._-]+$ ]]` guard before the heredoc.
- Real DigitalOcean IP `143.198.xxx.xxx` in usage examples →
  RFC 5737 `198.51.100.42`.

MEDIUM — code quality:

- `terraform.tfvars.example` had unrendered Copier jinja stubs
  (`{{ project_name }}`, `{{ enable_skinny_backups | lower }}`, etc.) —
  operators copying the file to terraform.tfvars would get literal jinja
  that terraform rejects. Replaced every stub with a real example value
  matching the variable's default in variables.tf.
- `.gitignore` ignored `.terraform.lock.hcl`, breaking provider
  reproducibility. Removed the ignore + added an explanatory comment.
- `versions.tf` header still said `# {{ project_name }}` → `s004-anythingllm`.
- `monitoring.tf` declared three alert resources that always created
  regardless of `var.enable_monitoring`. Added `count = var.enable_monitoring ? 1 : 0`.
- `scripts/restore.sh` had dead REMOTE/BACKUP_NAME assignments at lines
  20-21, overwritten by the arg-count block. Removed.

Not addressed (would expand scope):

- Infisical Machine Identity rendered into user_data → terraform state.
  Same trade-off as signoz-docker; needs a remote encrypted backend +
  ideally one-time-use bootstrap secret retrieval.
- `lifecycle { ignore_changes = [user_data] }` means cloud-init fixes
  require `tofu taint` to deploy. Documented behavior, not changed here.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(infra): adopt Path C + Layer 2 in s004-deployment; document migration

Resolves the two architectural issues deferred from PR #26 / PR #31:
  1. Infisical Machine Identity persisted in terraform state / DO metadata
     (Layer 2 bootstrap-secret rotation).
  2. `lifecycle { ignore_changes = [user_data] }` silently swallowing
     cloud-init updates (Path C: thin cloud-init + ansible app layer).

Reference implementation lives in s004-deployment/. Repo-wide pattern and
per-project migration checklist live in docs/INFRA_BOOTSTRAP_PATTERN.md.

Layer 2 — bootstrap-secret rotation (s004-deployment/):

- New rotate-bootstrap-secret.sh embedded in cloud-init `write_files`. Runs
  once via runcmd at first boot. Flow:
    1. Log in to Infisical with v1 (from terraform.tfvars → state → cloud-init).
    2. Decode JWT to extract identityId (base64url-safe, handles padding).
    3. POST /api/v1/auth/universal-auth/identities/{id}/client-secrets → v2.
    4. Atomically swap .infisical-auth.env to v2 after verifying it auths.
    5. POST .../client-secrets/{v1Id}/revoke to disable v1.
    6. Touch .rotation-complete (idempotent on re-run).
  Net effect: v1 in terraform state + DO droplet metadata is revoked within
  minutes of provisioning. v2 only ever lives on the droplet filesystem.
  Best-effort with structured logging — if the Machine Identity lacks
  self-management permission, the script logs cleanly to
  /var/log/s004anythingllm-rotation.log and the operator follows a manual
  runbook in s004-deployment/README.md.

Path C — thin cloud-init + ansible (s004-deployment/):

- Slimmed terraform/templates/cloud-init.yaml. It now handles ONLY first-boot
  bootstrap: package + Docker install, Infisical CLI install (artifacts-cli
  apt repo), .infisical-auth.env write, Layer 2 rotation, .bootstrap-complete
  marker, unattended-upgrades. Removed: compose.yaml, Caddyfile, backup.sh,
  daily-backup cron, infisical helper, docker pulls, docker compose up.
- New ansible/deploy.yml owns the app layer. Uploads compose + Caddyfile +
  backup.sh, renders the daily backup cron, pulls images, runs `docker
  compose up -d --remove-orphans`, waits for AnythingLLM health. Pre-tasks
  assert INFISICAL_PROJECT_ID is set and .bootstrap-complete exists.
- scripts/deploy.sh rewritten as a thin wrapper around `ansible-playbook
  ansible/deploy.yml -i 'host,'`. Idempotent — re-runnable any time
  compose/Caddy/scripts change, without touching terraform.

Layer 1 — DO Spaces remote state backend (already standard elsewhere):

- Added terraform/backend.tf + terraform/init.sh + spaces_* vars. State is no
  longer a plain local JSON file. Same pattern as
  keycloak-docker/sites/sso.weown.dev/ and signoz-docker (PR #26). Required
  for Layer 2 to actually reduce risk — without it the v1 in state lives on
  the operator's workstation indefinitely.

Documentation + migration plan:

- New docs/INFRA_BOOTSTRAP_PATTERN.md (cross-cutting). Explains both
  problems, both solutions, failure modes, compliance mapping (NIST CSF
  2.0, CIS Controls v8, ISO/IEC 27001:2022), per-project migration
  checklist for signoz-docker / searxng-docker / anythingllm-docker /
  keycloak-docker / wordpress-docker, a "what Layer 2 does NOT solve"
  section, and a future-hardening section (Vault wrapping, cloud-IAM
  mediation, systemd-credentials + TPM).
- Each *-docker README has a MIGRATION PENDING / MIGRATION PARTIAL banner
  pointing at the pattern doc + s004-deployment reference impl. Created
  searxng-docker/README.md (was missing).
- s004-deployment/README.md: rewrote Quick Start to reflect Path C
  workflow, added "Updating the deployment" matrix, added manual rotation
  runbook.
- s004-deployment/CHANGELOG.md: Unreleased entry.

Migration sequencing recommendation (in INFRA_BOOTSTRAP_PATTERN.md): each
*-docker project gets its own focused follow-up PR, order by deployment
criticality (anythingllm → wordpress → keycloak → signoz → searxng).
Each is ~2-4h plus an operator-scheduled tofu taint + tofu apply against
existing droplets. Not changed in this commit: the *-docker template
implementations themselves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(infra): per-project Migration Status sections + audited status table

Replaces the one-line "MIGRATION PENDING" banners with a per-project
Migration Status section in each *-docker/README.md. Each section is a
table showing the project's current Layer 1 / Layer 2 / Path C / Infisical
CLI state with citations to the files that need work, plus project-specific
notes. The shared 6-step migration checklist + rationale stays in
docs/INFRA_BOOTSTRAP_PATTERN.md (single source of truth).

Audit findings (2026-05-25):

- anythingllm-docker: Layer 1 missing, Layer 2 missing, Path C not
  adopted (no ansible playbook exists), legacy Infisical install.
  **Highest priority** — source of live deployments.
- wordpress-docker: Layer 1 missing, Layer 2 missing, Path C partial
  (ansible exists with compose + Caddy + Wordfence upload, cloud-init
  also embeds the app layer), legacy Infisical install.
- keycloak-docker: Layer 1 partial (template has backend.tf.jinja but
  no init.sh.jinja; the rendered sites/sso.weown.dev/ subdir has both),
  Layer 2 missing, Path C partial (site.yml.jinja with
  community.docker.docker_compose_v2 — most ansible-shaped of any
  template, but cloud-init still embeds app layer), legacy Infisical
  install.
- signoz-docker: Layer 1 done (PR #26), Layer 2 missing, Path C partial
  (ansible exists, cloud-init also embeds), legacy Infisical install
  (with open ZooKeeper anon-login accepted-risk note).
- searxng-docker: Layer 1 done (PR #26), Layer 2 missing, Path C partial,
  legacy Infisical install.

Updated docs/INFRA_BOOTSTRAP_PATTERN.md status table to reflect the audit
(was less precise — said "Pending" without distinguishing Layer 1 vs Path C
state). The migration sequencing recommendation in the doc unchanged:
anythingllm → wordpress → keycloak → signoz → searxng.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(infra): migrate anythingllm-docker to Path C + Layer 2; relocate s004; add auto DO tagging

Big bundle: completes Option B (migrate the anythingllm-docker template to
the canonical bootstrap pattern, then relocate s004 as a generated site of
that template), AND adds the automatic DO droplet tagging system across
feature scripts + ansible.

Template migration (anythingllm-docker/template/):

- terraform/backend.tf.jinja + terraform/init.sh.jinja added (Layer 1 — DO
  Spaces remote state with SSE-C, same pattern as signoz-docker).
- terraform/templates/cloud-init.yaml.jinja replaced with the slim Path C
  version + embedded rotate-bootstrap-secret.sh (Layer 2). Cloud-init now
  handles only: package install, Docker install, Infisical CLI install
  (artifacts-cli apt repo — current channel), .infisical-auth.env write,
  bootstrap-secret rotation, .bootstrap-complete marker, unattended-upgrades.
  Compose, Caddy, backup script, cron all removed — they live in ansible now.
- terraform/main.tf.jinja: `replace('-', '_')` (was `replace('-', '')` —
  caused the volume-name mismatch we hit in s004), SSH firewall rule uses
  `var.ssh_source_cidrs`, `lifecycle { ignore_changes = [user_data, tags] }`
  so runtime tag mutations stick across `tofu apply` cycles.
- terraform/variables.tf.jinja: adds ssh_source_cidrs + spaces_* trio.
- terraform/terraform.tfvars.example: adds the new vars with placeholders.
- terraform/monitoring.tf.jinja: `count = var.enable_monitoring ? 1 : 0`
  on each alert so the variable actually gates resources.
- docker/compose.prod.yaml.jinja: image pinned via copier vars, Caddy
  /var/log/caddy bind mount so otel-agent's filelog/caddy receiver can read
  access logs.
- docker/Caddyfile.jinja: domain substitution via copier var.
- scripts/deploy.sh.jinja: thin ansible-playbook wrapper. Requires
  INFISICAL_PROJECT_ID env var.
- scripts/backup.sh.jinja: hardcoded volume names use
  `{{ project_name | replace('-', '_') }}_storage` etc. Wraps the
  `docker ps --format '{{ .Names }}'` lines in {% raw %}{% endraw %} so
  copier doesn't try to interpret the Go template syntax.
- scripts/restore.sh.jinja: same volume fix + BACKUP_NAME regex validation
  before the SSH heredoc + INFISICAL_PROJECT_ID env requirement.
- ansible/deploy.yml.jinja added (Path C app layer): uploads compose +
  Caddyfile + backup.sh, renders daily cron + logrotate, pulls images,
  reconciles compose stack, waits for health, then updates DO tags
  (skinny-backup + commit-<sha>) via the shared helper.
- copier.yaml: adds ssh_source_cidrs variable (json type).
- README.md.jinja: rewrote Quick Start for Path C workflow.
- .gitignore: stop ignoring .terraform.lock.hcl.

s004-deployment/ → anythingllm-docker/sites/s004/:

- `git mv` preserves history per file. Then overlaid the freshly-rendered
  template output (via `copier copy --data-file s004-answers.yaml`) so the
  site reflects exactly what the migrated template produces today, with
  underscore-consistent paths (/opt/s004_anythingllm/, s004_anythingllm_*
  volumes) that were previously inconsistent.
- All cross-references in INFRA_BOOTSTRAP_PATTERN.md + the 5 *-docker
  READMEs updated from s004-deployment/... → anythingllm-docker/sites/s004/...

Auto DO tagging (new):

- scripts/tag-droplet.sh — shared helper for tag mutation. Subcommands:
  add, remove, replace-prefix, set-commit, list. Chainable in a single call.
  Resolves droplet name → ID via doctl, mutates tags via
  `doctl compute droplet-action tag/untag --wait`. Idempotent.
- scripts/bootstrap-otel-agent.sh: after each successful bootstrap, looks
  up the droplet name from the target IP and adds the `otel` tag.
- scripts/deploy-otel-fleet.sh: same — adds `otel` after each successful
  deploy.
- anythingllm-docker/ansible/configure-allm.yml: after MCP config changes
  successfully apply, looks up the droplet from inventory_hostname and
  adds the `searxng-mcp` tag.
- anythingllm-docker/template/ansible/deploy.yml.jinja: on every deploy,
  invokes tag-droplet.sh with `replace-prefix commit- commit-<sha> add
  skinny-backup`. Restoring "the last working deployment" becomes read the
  tag, `git checkout <sha>`, re-run deploy.sh.
- terraform/main.tf.jinja `ignore_changes = [tags]` so feature/state tags
  added at runtime aren't reverted by subsequent `tofu apply` runs.

Documentation (docs/INFRA_BOOTSTRAP_PATTERN.md):

- New "DO tag taxonomy" section documenting the three layers: project tags
  (terraform-set), feature tags (script-driven — otel, skinny-backup,
  searxng-mcp), state tags (commit-<sha>, replaced each deploy).
- Project migration status table updated: anythingllm-docker promoted to
  reference implementation; wordpress now priority 1 follow-up.

Verified the migrated template renders cleanly via `copier copy` — produces
the 17 files that match anythingllm-docker/sites/s004/ in structure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(anythingllm-docker): Copilot round-4 findings on PR #31 ab65db6

Fixes 7 real findings from Copilot's review of the migration commit (other
~15 were stale — looking at pre-overlay file state or pointed at the
deferred Layer 2 limitation already documented). Each fix applied to the
TEMPLATE; `sites/s004/` then re-rendered via `copier copy` to keep both
in sync.

Real fixes:

- `template/docker/Caddyfile.jinja`: was `output stdout` while the compose
  service bind-mounts `/var/log/caddy:/var/log/caddy` — the mount was
  useless. Switched to
  `output file /var/log/caddy/{{ project_name }}.log` so the file lands
  on the host where otel-agent's `filelog/caddy` receiver can read it
  (and survives container recreation).
- `template/README.md.jinja`: backup-storage and migration-procedure
  examples used `/opt/{{ project_name }}/backups/` (hyphenated, e.g.
  `/opt/s004-anythingllm/backups/`) but the actual deployment path is
  `/opt/{{ project_name | replace('-', '_') }}/backups/` (underscore).
  Updated to underscore form everywhere paths appear.
- `template/README.md.jinja`: restore example was
  `anythingllm-ai_backup_20260115_120000` (stale placeholder from the
  pre-migration template). Updated to `{{ project_name }}_backup_…`.
- `template/scripts/backup.sh.jinja`: remote-mode ran
  `ssh "$host" "$BACKUP_CMDS"` directly — `SPACES_ACCESS_KEY` /
  `SPACES_SECRET_KEY` were NOT in the inner shell's env, so the
  `aws s3 cp` upload step would silently no-op. Wrapped the remote bash
  in `infisical run --projectId=… --env=…` (sources
  `/opt/<project>/.infisical-auth.env` the same way restore.sh does).
  Added `INFISICAL_PROJECT_ID:?` guard so remote mode fails fast.
- `template/terraform/templates/cloud-init.yaml.jinja`: re-added `awscli`
  to the packages list. The slim cloud-init had dropped it during the
  Path C refactor, but the daily backup cron uses `aws s3 cp` so without
  awscli the first cron tick fails.
- `template/scripts/deploy.sh.jinja`: `ansible-galaxy collection install
  community.docker` was unpinned — operators on different workstations
  could end up with different versions. Pinned to `==3.13.0` with a
  precise version check.

Re-rendered `anythingllm-docker/sites/s004/` via `copier copy
anythingllm-docker/ <tmp> --data-file s004-answers.yaml --defaults
--trust`, then overlaid the rendered output onto sites/s004/.

Stale/not-fixed comments (acknowledged):
- Several "volume name mismatch" comments were on the OLD code — current
  scripts use `s004_anythingllm_storage` matching compose.
- `terraform/main.tf` "Infisical secret in user_data → tfstate" is the
  deferred Layer 2 limitation (rotation runs at first boot; see
  docs/INFRA_BOOTSTRAP_PATTERN.md "What Layer 2 does NOT solve").
- `terraform/versions.tf` "no remote state backend" — backend IS in
  backend.tf (separate file). Terraform merges multiple `terraform {}`
  blocks across files. False positive.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(anythingllm-docker): Copilot round-5 — fix HCL blocker + 8 other real issues

Resolves the 9 actual bugs from Copilot's review of `3e312a4` (other ~19
comments triaged as either stale or false-positives — captured below).

CRITICAL — tofu blocker:

- `template/terraform/variables.tf.jinja:58` (and tfvars.example): the
  `ssh_source_cidrs` default was `{{ ssh_source_cidrs }}`, which Copier
  renders as Python's list-repr `['0.0.0.0/0', '::/0']` (single quotes) —
  NOT valid HCL. `tofu plan` would error on "Invalid character". Switched
  to `{{ ssh_source_cidrs | tojson }}` so Copier emits the JSON form with
  double quotes (which HCL parses as a list).

Files-not-being-rendered-by-Copier blocker:

- `_templates_suffix` defaults to `.jinja` in Copier, so files without
  that suffix in template/ were copied as-is — including their `{{ ... }}`
  template syntax. The rendered s004 site had unrendered placeholders in
  `.gitignore`, `terraform/versions.tf`, and `terraform/terraform.tfvars.example`.
  Renamed all three to add the `.jinja` suffix.

Real correctness bugs:

- `anythingllm-docker/ansible/configure-allm.yml`: the tag step did
  `ip="{{ inventory_hostname }}"` then looked up the droplet by PublicIPv4.
  With the documented `-i 'root@<ip>,'` inventory pattern,
  `inventory_hostname` is `root@<ip>`, not `<ip>`, so the doctl lookup
  always failed silently. Added `ip="${target##*@}"` to strip the prefix.
- `template/scripts/backup.sh.jinja` remote-mode: was
  `bash -c '$BACKUP_CMDS'` where BACKUP_CMDS contains the literal
  `'table {{.Names}}...'` directives. Outer single quotes break on the
  inner single quotes. Restructured to invoke the droplet's own
  `/opt/<project>/backup.sh` (uploaded by ansible) inside `infisical run`.
- `template/scripts/restore.sh.jinja` remote-mode: same quoting issue +
  sourced `/opt/<project>/infisical-auth.sh` (a script that doesn't
  exist; cloud-init writes `.infisical-auth.env`). Restructured the same
  way: source `.infisical-auth.env`, run `infisical login`, then exec
  `/opt/<project>/restore.sh "$BACKUP_NAME"` under `infisical run`.
- `template/terraform/main.tf.jinja:36` tag-comment listed
  `anythingllm/ansible/configure-allm.yml` (no such file); the playbook
  lives at `anythingllm-docker/ansible/configure-allm.yml`. Fixed.
- `template/docker/compose.prod.yaml.jinja`: `LLM_PROVIDER` and `VECTOR_DB`
  were hardcoded to "openrouter"/"lancedb" while the template still
  prompts for `llm_provider` and `vector_db` Copier vars (and passes them
  through Terraform). Those Copier knobs were dead. Replaced with
  `{{ llm_provider }}` / `{{ vector_db }}`.

Stale/false-positive comments (verified, not fixed):

- Caddyfile/compose hardcoded `s004.ccc.bot` — correct for the SITE.
- main.tf:25 "Infisical secret in user_data" — deferred Layer 2.
- versions.tf:6 "no remote state backend" — false positive; backend IS
  in backend.tf (separate file).
- CHANGELOG anythingllm_image — already `reg.mini.dev/anythingllm:1.7.2`.
- cloud-init jq `$$V2` — terraform escape, becomes `$V2` post-templatefile
  inside single-quoted jq script (jq's `--arg V2` definition).
- Volume name mismatch comments — scripts already use the explicit
  `s004_anythingllm_storage` matching compose `name:` declarations.

Re-rendered sites/s004/ from the updated template via copier; all 9 fixes
verified in the rendered output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pr31): independent-review findings — restore auto-tagging + harden rotation script

Final pre-merge pass from an independent code-review agent caught one HIGH
bug that broke the auto-tagging feature this PR was meant to add, plus
three LOW hardenings. Applied to the template and re-rendered sites/s004/.

HIGH — auto-tagging was silently dead-on-arrival:

  - `template/ansible/deploy.yml.jinja:202` used
    `playbook_dir | dirname | dirname | dirname` to find scripts/tag-droplet.sh.
    But the layout is:
      <repo-root>/anythingllm-docker/sites/<name>/ansible/deploy.yml
    So three dirname operations land at <repo-root>/anythingllm-docker/ —
    not the repo root. With `failed_when: false`, the broken tagging step
    never failed, so `skinny-backup` and `commit-<sha>` tags silently
    never appeared on droplets. Added one more `| dirname` to walk to the
    repo root. The sibling task in anythingllm-docker/ansible/configure-allm.yml
    is one level shallower and was already correct (dirname × 2).

LOW — rotation-log hardening:

  - cloud-init.yaml.jinja was logging the full Infisical API response on
    v2-mint failure. Replaced with `jq -r '.message // .error // .'`
    (head -c 500) so any future API change that echoes request fragments
    doesn't persist the bearer token.
  - Pre-chmod the rotation log to 0600 BEFORE writing so it's never
    world-readable even on failure paths.

LOW — config knob clarity:

  - `INFISICAL_HOST="$${INFISICAL_HOST:-https://app.infisical.com}"`
    looked overridable, but cloud-init runcmd: has a minimal env so the
    override couldn't actually be set externally. Dropped the indirection
    and inlined the SaaS URL.

LOW — tag-droplet.sh duplicate-name safety:

  - Previously did `awk … {print $2; exit}` — first-match-wins on
    duplicate droplet names. Now counts matches and errors out cleanly
    if 0 or >1.

Reviewer findings explicitly NOT fixed (purely cosmetic):
  - U+21BA arrow in usage text (Finding 6)
  - Per-project README status drift risk (Finding 5)

Re-rendered sites/s004/ via copier; HIGH fix verified in rendered
ansible/deploy.yml, LOW fixes verified in
sites/s004/terraform/templates/cloud-init.yaml.

This is the merge-ready state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pr31): Copilot round-6 — 3 real bugs + 1 cosmetic

Triaged 25 Copilot comments on `0993c43`. Verified ~20 as stale (Copilot
re-flagged from older state — current code is correct). 4 actionable:

REAL (would break first deploy):

- `template/ansible/deploy.yml.jinja:156` — was using
  `community.docker.docker_image`, which requires the python3-docker
  Python SDK on the target host. The slim cloud-init only installs
  Docker itself, not the SDK. Switched to
  `community.docker.docker_image_pull` (CLI-based — no SDK needed).
  Available since community.docker 3.4; we pin 3.13.0.

REAL (silently-broken config knob):

- `template/scripts/backup.sh.jinja` + `restore.sh.jinja` — were hardcoding
  `REMOTE_STORAGE="do-spaces"` while `copier.yaml` offers
…